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Method and Apparatus for Interaction with Broadcast Television Content 



This invention relates to a method and apparatus by which television (TV) viewers are able to 
interact with the broadcast signal on their screen. 

5 

Traditional TV broadcast is characterised by temporal synchronisation for all viewers, 
ignoring the small differences in transit times of the signal due to differences in transmitter to 
user distances, that is at the moment of broadcast all viewers receive the same, uniform 
signal substantially instantaneously. Interactive forms of the medium, such as Two Way TV, 

10 Web TV are generally characterised by providing the viewer the facility to interact with a 

designated interactive area of the TV screen. By way of example only, this may utilise a spirt 
screen in which the interactive area may have Internet information, an on screen menu or 
information which by way of example only may be relevant to the main broadcast part of the 
TV screen. These systems are characterised by presenting information in response to the 

15 user's interaction within a fixed predefined interactive area of the screen. 

By way of illustration, and in contrast to limited interaction with TV broadcast on current 
systems, computer games consoles for example but not exclusively the Sony Playstation, 
Sega Saturn, Nintendo 64 are presented to the viewer on TV screens or dedicated display 

20 screens, but each viewer has the ability to achieve unique interaction with the imagery/ 
sounds presented on the TV, which are processed by the games console using computer 
technology, in response to the users actions. Generally the user's perceived TV environment 
including but not exclusively images, sounds and other sensory signals are produced by 
computer generation within the games console, which may or may not also utilise digitised 

25 predefined data, such as sounds in the TV environment which shall be referred to as the 
computer generated environment (CGE) or in the specific case of images, computer 
generated imagery (CGI). These games consoles and the CGE are characterised by high 
frequency update rate, by way of example only in excess of 25 frames per second or 25 Hz, 
which the gives the impression of instantaneous or real time response to the user's actions 

30 and also a smooth and seamless dynamic image in which the individual frames are not 
discernible as individual frames, rather they contribute to the overall real time dynamic 
environment, giving the impression of real time control and interaction with the CGE. 

It is desirable that the real time CGE is as realistic as possible, and greater degrees of 
35 realism are generally achieved by increased computer processing power and by using the 
most efficient representation in terms of realism versus processing power. By way of 
example only, one of the most efficient representations for CGI uses relatively coarse 
polygonal or faceted geometric model in which the greatest detail in terms of polygon 
distribution would generally be used in the more geometrically complex areas. By a process 
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known as texture mapping, in which photorealistic textures representing surface features are 
mapped onto the individual polygon faces, a relatively realistic CGI is achieved 
notwithstanding the relatively coarse polygonal geometry representation. The product of the 
number of texture mapped polygons in the CGI and the image update rate measured in 
5 frames per second yields a number defining the number of texture mapped polygons the 
computer processing is required to process per second, which may by way of example only 
be 1 ,000,000 polygon per second. 

We have identified that the level of interaction offered to the viewers of interactive broadcast 
10 TV is limited, and this consequently limits the applications of such medium. Furthermore we 
have identified that although the interaction of a games console Is greater than interactive 
TV, the actual theme of the CGE is limited to the specific game content loaded into the 
console, by way of example only via CD ROM, cartridge, the Internet or broadcast to a TV 
with the appropriate hardware to run CGE type games or by other means. 

15 

For the avoidance of doubt we recognise that systems are proposed for interactive TV, and 
that games content can be broadcast, however the present invention relates to a capability to 
interact with the actual broadcast itself as it appears on the TV screen in real time. 

20 According to the present invention a method and equipment is provided for use in association 
with a TV set to provide levels of interaction with broadcast content that is normally 
associated with a games console, whereby such interaction is achieved within the main 
broadcast as presented on their TV screen. By way of example only, this provides the user 
or viewer with a CGE, hereinafter referred to as the foreground computer generated object 

25 (foreground CGO) which interacts with the full screen broadcast hereinafter referred to as the 
background object, which by way of example only may be delivered by Digital Terrestrial. 
Satellite or Cable broadcast medium and in which the background object changes 
continuously during the broadcast transmission, not only from programme to programme but 
from frame to frame at a rate of, by way of example only, in excess of 20 Hz. 

30 

For the avoidance of doubt, this invention offers full screen interaction via the foreground 
CGO with objects represented in the broadcast, for example but not exclusively visible 
objects, and such interaction is assured even though objects may change position, shape, 
motion, behaviour at the broadcast update rate, and furthermore the nature of the interaction 
35 with such objects may also change at the broadcast rate. 

The term "broadcast" as used herein is intended to cover all transmission medium, including 
but not exclusively, digital terrestrial, cable, satellite, broadcast to all display devices, 
including but not exclusively, TV, computer, games console. The term "channel" as used 
40 herein refers to different broadcast content relating to the same programme, and all channels 



• are broadcast substantially simultaneously. The term "programme" refers to a set of such 
multiple channels which relate to the same programme, where each channel within the set of 
channels making up the programme broadcast are temporally synchronised, and relate to the 
same content but, by way of example only, offer an altemative view of the content. The 
5 term "CGE" as used herein is intended to cover a multi media representation, including but 
not exclusively still images, dynamic images, sounds, real time images and real time audio 
signals. The temri "foreground CGO", "foreground CGE" or "foreground" as used herein is 
intended to any and all representations which are not part of the broadcast, but are computer 
generated and which may be displayed on the TV screen, by way of example but not 
10 exclusively overlaid on the broadcast or in a separate interactive area of the screen. 

Alternatively the "foreground CGO", "foreground CGE" or "foreground" may apply to such 
representations which are not part of the broadcast and are not displayed or othenArise 
represented on the TV screen. By way of example only, portions or sections of the 
foreground CGO may be hidden from the user or viewer for the purpose of acting as 
15 geometric reference to calculate interaction between the foreground CGO and the broadcast. 
The tenm "computer " in the context of "computer generated", "computer processing", 
"computer generated imagery" or computer generated environment" refers to any apparatus, 
equipment, hardware, software, parts thereof and combinations thereof which provides the to 
process the foreground CGE, and by way of example only may be a computer, a set top t>ox 
20 (as produced by General Instruments, Pace Micro Technology by way of example only), a 
games console (as produced by Sony. Sega, Nintendo by way of example only), parts or 
sections thereof, or customised hardware including but not exclusively computer memory, a 
processor and an optional graphics processor. The tenm "TV screen" or "screen" or "display 
screen" as used herein is intended to cover any display device or system or assembly in 
25 which there is a display element including but not exclusively. TV screen, computer monitor, 
projection system, head mounted . The term "receiving hardware" as used herein refers to 
any apparatus, equipment, hardware, software, parts thereof and combinations thereof which 
receives the broadcast channels, the receiving hardware input, from the broadcast medium 
and transmits the signals, the receiving hardware output, to the screen, and may by way of 
30 example only be referred to as a decoder, and by way of example only may be a computer, a 
set top box (as produced by General Instruments, Pace Micro Technology by way of exarnple 
only), a games console (as produced by Sony. Sega, Nintendo by way of example only), 
parts or sections thereof, or a customised hardware including computer memory, a processor 
and an optional graphics processor. The tenm "nominal user posKion" or "user position' 
35 defines a distance measured normal from the plane of the TV screen in a normal direction, to 
the position of the user, where this distance and the user position are used purely for 
calculation purposes, and impose no further restriction on the actual user position in addition 
to the everyday physical constraints. The tenms "viewer^, "viewers", "user* and "users" can 
be exchanged and interchanged with no loss of generality. The term "controller" or "hand 
40 controller" as used herein refers to any device with which the foreground CGO and 
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interaction with the background CGO is controlled by the user, including but not exclusively 
motion and interaction, requests to upload and download other data or information. By way 
of example only, the controller may be an infrared device operated by buttons or direct voice 
activation. 

5 

Advantageously, with the broadcast signal, there are multiple channels relating to the 
programme, each channel representing, by way of examples only, an alternative view, a user 
selectable view, additional relevant information pertinent to the conresponding frame on one 
or more of the other channels, where each channel updated at, by way of example only. 
10 25Hz. 



Advantageously, within the broadcast signal, control parameters are transmitted on each 
channel, and temporally synchronised with each frame of the channel broadcast, which may 
by way of example only be transmitted at a rate of 25 frames per second, and such control 
1 5 parameters define the interaction the user can achieve between the foreground CGO and the 
background CGO or the broadcast for each frame, and those control parameters and the 
associated defined interaction may vary at a rate, of for example only, 25 Hz. The control 
parameters are used by the computer to define the interaction for that particular frame of the 
channel broadcast. 

20 

According to the present invention, the signal is broadcast containing multiple channels 
relating to the same programme, and temporally synchronised control parameters. The 
broadcast is received by the receiving hardware, which by way of example only may be a set 
top box or part thereof. The receiving hardware transmits the decoded broadcast images to 
25 the TV screen at the predetermined rate, by way of example only, at 25Hz. 

Advantageously, the control parameters are concurrently or simultaneously received by the 
computer, and for each frame of the broadcast signal the areas of interaction and the nature 
of such interaction is stored for each frame. By way of example, the areas of interaction are 

30 defined by the co-ordinates in three orthogonal axes resolved or projected into the 

substantially 2 dimensional plane of the TV screen with a predefined viewing angle, resulting 
in a 2 dimensional representation of the interaction areas overtaid on each frame of the 
broadcast and such frames and the associated interactive areas may change at the update 
rate, which by way of example only, is 25 Hz. By way of example only the viewing angle 

35 may be 90** in a horizontal plane, which in turn defines a nominal user position for computer 
calculations and projection from the three dim'ensional representation to the two dimensional 
screen representation. The viewing angle and the conresponding nominal user position define 
the extent of the broadcast background visible on the screen, and are preferably selected to 
complement and match broadcast standards for visible viewing angle. 



• Advantageously the nature or the interaction tor each area may include, but not exausiveiy, 
areas that the user controlled foreground CGO can interact with, areas which the user can 
click with a cursor device, areas which are linked to infonmation sources activated by clicking, 
the type of interaction between the foreground CGO and the interactive area. 

5 

Advantageously, the processing within the computer generates the foreground CGO, which is 
the user controlled representation by which he user interacts with the interactive areas 
defined by the control parameters for each frame. The computer transmits the foreground 
CGO to the screen at a rate of 25H2 by way of example only, to be overiaid on the broadcast 
1 0 signal for each frame of the broadcast, whereby the position, shape and other features of the 
foreground CGO including but not exclusively colour, sound, direction of motion, visibility, as 
modified by the user interaction, and updated at a the aforementioned rate. 

Preferably, within the TV screen, there is apparatus to mix the representations of the 
15 background broadcast and the foreground CGO such that the foreground CGO is overlaid on 
the background broadcast and the background CGO. 

Advantageously, within the mixing apparatus there is processing available to determine 
which individual pixels within the foreground CGO, the background broadcast and any other 

20 displayed feature are closer to the plane of the TV screen when measured in the 

aforementioned three orthogonal axes defining a three dimensional geometric space. 
Furthemiore, for each pixel position of the screen measured in the two dimensional screen 
co-ordinates, the pixel properties including but not exclusively colour, are those representing 
the properties of the object closest to the screen measured in the aforementioned 3 

25 orthogonal axes at the point projected onto the two dimensions of the TV screen. 

Advantageously, geometric information relating to the background is conveyed in the control 
parameters defining the background CGO, Advantageously, parts or sections of objects 
including but not exclusively the foreground CGO, the background CGO and the broadcast 
background which when resolved into the screen co-ordinate system lie outside the screen 

30 dimensions, or lie between the screen and the nominal user position, or are within the screen 
dimensions but further away from the screen than some other object, are not displayed. 
Advantageously, this process is referred to as culling. 

Advantageously for material recorded in any format for subsequent broadcast, including but 
35 not exclusively Betacam or digitally stored images, the control parameters are encoded in or 
with the material prior to broadcast. The control parameters for each frame of the broadcast 
include but not exclusively the areas of interaction defined in the two dimensional screen co- 
ordinates, the nature of the allowable interaction, the resulting action arising from such 
interaction. By way of example only, interaction with one of the areas may cause infonmation 
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to be displayed, or an alternative channel of the broadcast to be displayed, or further 
information be displayed on the screen. 

Advantageously for live broadcast material or real time feeds, the control parameters are 
5 generated automatically or semi automatically utilising, by way of example only, vision 

systems which interpret each frame of the broadcast as it occurs in order to identify particular 
features within the frame image, and within such group of features areas that are to be 
automatically converted to interactive areas. The conversion from vision system identified 
features to interactive areas may by augmented by the use of technology including but not 

10 limited to, artificial intelligence, neural networks, knowledge based systems or combinations 
thereof, to generate such interactive areas and other control parameters subject to 
predefined rules based on the identification of the nature of the feature. Preferably, the rate 
at which the control parameters are generated is commensurate with live broadcasting, and 
by way of example only, this will be at a rate equivalent to the broadcast frame update rate, 

15 and way of example only, 25H2. For the purpose off illustration only, this technique used 
with live broadcast may be used for computer games based on live footage or training 
simulators based on live geographic features and scenarios. 

Advantageously, the type of foreground CGO may be, by way of example only, selected by 
20 the user from a library of such foreground CGO stored in the computer memory, or on a data 
storage device connected to the computer which by way of example only may be a CD ROM, 
or a Digital Versatile Disc (DVD). Preferably the foreground CGO may also be downloaded 
to the computer at the beginning or during the broadcast of the control parameters. Such 
definition of the foreground CGO will define features including but not exclusively, colour, 
25 shape, texture, allowable movement, sound effects, articulation on screen. 

Advantageously, the foreground CGO object movement and interaction is controlled by way 
of example only, via a user input using a hand controller which, by way of example, may be a 
standard or modified infrared controller, and subject to the rules defied by the control 
30 parameters for each frame. Altematively and preferably, other input devices may be used to 
control the foreground CGO and interaction including but not exclusively, voice activation, 
mouse, game controller or pad. 

Advantageously, the nature of the foreground CGO will match the nature of the broadcast. 

35 By way of example only, a broadcast comprising a road or driving theme where the road 
representation itself is an interactive area may suit a foreground CGO based on a vehicle, 
such as a car, and such car based foreground CGO will have motion dynamics 
representative of a real car, to the extent that a games console car is representative. By way 
of a further example, a broadcast comprising an exploration or tour theme may suit a 

40 foreground CGO based on a based on an articulated walking human figure, and the control 
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foreground CGO through doors in the broadcast background, but not through walls. 



Advantageously, in some broadcast programmes or themes, the foreground CGO may not 
5 match the broadcast, and may be by way of example, an abstract representation of the 
position of the foreground CGO, such as a cursor, an arrow, an icon graphic of a hand. 

According to the present invention when the user controls the foreground CGO with the 
controller, the foreground CGO may perform functions including but not exclusively, 
10 translational motion and rotational motion about the three orthogonal axes and combinations 
thereof resolved into the two dimensional screen co -ordinates, initiate sounds, interaction 
with the defined interactive areas where such interaction results in further action, by way of 
example only, infonmation presentation in text, graphic, video or multi media fonms or 
combinations thereof. 

15 

Advantageously, the user has the option of downloading information from the broadcast 
relevant to the broadcast, by way of example only, foreground CGO representations, 
additional information, software, control parameters. 

20 Preferably the user will also have the option of uploading infonmation relevant to the 

broadcast programme, by way of example only uploaded requests for infonmation or actions 
initiated by specific interaction between the foreground CGO and the background CGO, and 
where such uploading is achieved by way of example only by communication between the 
controller and the computer. By way of further examples, certain types of uploaded request 

25 may be considered as an uploaded control parameter, which by way of example only causes 
a switch to a different channel of the broadcast containing additional infonmation, whereas 
other uploaded requests may be for information not available within the broadcast, and such 
requests are routed, by way of example only, to a Worid Wide Web site for the specific 
programme via telephone connection and modem. Preferably the user will have the option 

30 of uploading data relevant to the programme, by way of example only, perfonmance scores 
achieved by the user in an interactive game scenario. 

By way of further explanation, it may be informative to consider he control parameters and 
the interactive areas defined by such parameters as a background CGO with which the user 

35 via the controller can cause the foreground CGO to interact with. Preferably, but not 
essentially, the background CGO includes a coarse geometric representation and visual 
quality is greatly improved by the aforementioned technique of texture mapping whereby the 
broadcast image for each frame is a substantially full screen texture which we will refer to a 
Supertexture: Preferably, the geometry of the background CGO is not visible to the user. By 

40 way of further clarification the combination of the foreground CGO, the interaction defined In 
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the control parameters, the background CGO defined by the control parameters and the 
broadcast Supertexture provide an interactive CGE based on the broadcast technology which 
is comparable with that achieved with a games console. 

5 According to this invention, the aforementioned combination of features provides the user 
with the ability to interact with features within the broadcast. 

Advantageously, this invention may be used as the basis or foundation of a commercial 
service, by way of example only, in which the user pays for usage, by way of example only 
10 on a per programme or per unit time basis. Advantageously, such payment method may t>e 
incorporated into the apparatus, such as smart card operation, or an additional feature of the 
interaction via the controller, such as a user capability to enter credit card infonmation which 
is then treated as uploaded information as previously described, using secure transaction 
protocols and techniques. 

15 

Reference is now directed to the accompanying and following pages of text and drawings 
disclosing additional material with respect to the background and general factors concerning 
the present invention, including specific embodiments thereof. It is to be understood that in 
the present application, or any application claiming priority herefrom, the broad or broadest 
20 aspects of the invention are not to be understood to be limited by any detailed disclosure in 
the accompanying description, but individual features therefrom may be taken individually 
therefrom for use in combination with the concepts disclosed above in relation to the claiming 
of the broader aspects of the present invention. 

25 There now follows accompanying pages of text and drawings namely 1 page of text and 2 
pages of drawings. 
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Figure 1 shows a schematic of the apparatus and connectivity. The programme broadcast 
has an origin 1 and broadcasts the multiple channels 15 via the broadcast medium 7. which 
are received by the receiver hardware 2. The control parameters 10 are passed to the 
computer 3 . and the Initial channel is selected by the upload request signal 11 from the 
computer 3 to the receiver hardware 2. The selected channel 8 is decoded and transmitted 
to the mixer 4. The computer processes the foreground CGO and the background CGO and 
the calculated representation 16 is transmitted to the mixer 4. The foreground CGO. the 
t>ackground CGO. the broadcast channel 8 are combined for every pixel on the TV screen 5 
and the resulting combined signal 17 is transmitted to the TV screen 5. The multiple 
channels 15, the control parameters 10, the selected channel 8. the upload request signal 1 1 , 
the foreground and background CGO 16 and the combined signal are updated at a rate of, by 
way of example only. 25 times per second. The foreground CGO and interaction with the 
t)ackground CGO is controlled by the controller 6. Control signals from the controller are 
transmitted 9 to the computer, which by way of example only, may modify the foreground 
CGO representation whilst the background CGO is updated based on the updated broadcast 
control parameters 10 and the recalculated representation 16 is transmitted to the mixer 17. 
Uploaded information or requests for information not contained within the broadcast will be 
routed 12 to. by way of example only, a World Wide Web site for the specific programme via 
telephone connection and modem 18. In the case where the invention is used as the basis of 
a commercial service, the receiver hardware 2 is activated only after valid user payment has 
been made via the smart card device 21 . and the control activation signal 22 has been sent 
to the receiver hardware 2, thereby initiating the whole process 

Figure 2 shows a schematic of the components making up the combined signal 17, which are 
the broadcast channel 8 of which there may be, by way of example only. 25 per second, the 
component of the control parameters 10 which define the areas of interaction or the 
background CGO 10(a) which will be temporally synchronised with the broadcast channel 8, 
the foreground CGO definition which may have a component 10(c) from the control 
parameters 10 which will be temporally synchronised with the broadcast channel 8. plus 
additional definition 20 from, by way of example only, a CD ROM or other storage device 19, 
and the rules goveming interaction 10(b) from the control parameters 10 which will be 
temporally synchronised with the broadcast channel 8. 

Figure 3 shows the foreground CGO 10(c), 20 overlaid on the background CGO 10(a) and 
the broadcast channel 8, and this combines to form the combined signal 17, which may vary, 
by way of example only. 25 times per second. Figure 4 shows the foreground CGO 10(c). 20 
overiaid on the background CGO 10(a) and the broadcast channel 8. and the articulation and 
motion of the foreground CGO 10(c), 20 as controlled by the controller 9 via the computer 10 
to form the combined signal 17. This articulation, motion or other interaction may vary, by 
way of example only, 25 times per second. 
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